Using SalesforceDX to perform Bulk API operations

As noted the other day (Using SalesforceDX to automate getting Apex class test coverage percentages) SalesforceDX is great for many things but one of the ways is automate some operations that are time consuming or just takes a lot of manual work each time. One of these things are Bulk API operations which in of by themselves are not hard but there is no UI for them besides the DataLoader and no console API when using the DataLoader when not on Windows.

The customer I’m working for currently has a monster data load to perform and one of the things I’ve done is writing  script to split the data into data sets. One set per country – 91 sets all in all. All sets consists of 3 files to support the data load. One file for Accounts and two additional files for custom objects that needs to be loaded as well. All in all it’s a lot of clicking in the DataLoader and it doesn’t really scale for testing. That’s a lot of clicking in DataLoader when testing.

But I’m lucky as SalesforceDX receives new functionality all the time and at some point some data Bulk API features had snuck by me so I was pleasantly surprised to discover force:data:bulk:upsert and force:data:bulk:delete today. They we just what I needed. SalesforceDX to the rescue yet again…

So today I grabbed by IDE by the horns (#vscode in my case) and wrote some wrappers around the Bulk API capabilities of SalesforceDX. The fact that all SalesforceDX commands takes an optional –json argument makes it easy to script and parse responses. This combined with select-shell from npm I now have a nice CLI interface to doing Bulk data loads. The script looks as the available data sets and asks me what country to load data for and then what export timestamp to process (the data sets may exists in multiple versions). Then it goes and does its thing UPSERTing all 3 times in turn and reports status. So nice. The Bulk API is asynchronous so  the script also handles polling for job status and only proceeds once the job has completed successfully.

$ ./upsert mheisterberg@example.com.appdev
SFDX - Org for mheisterberg@example.com.appdev is connected...

Select country code:
 ae
 au
 ca
 cn
 es
 fr
 hk
 hu
 co
 ▸ it
 jp
 kr
 my
 pt
 sg
 th
 tr
 tw
 us

Select timestamp:
 2018-04-16T07:25:37Z
 2018-04-16T08:31:28Z
 ▸ 2018-04-16T08:34:14Z

Will process following data
Country : it
Timestamp: 2018-04-16T08:34:14Z
UPSERT for Account data...
Issued UPSERT bulk request to object (Account) - id 7516E000002DckQQAS, jobId 7506E000002QQ3zQAG - state: Queued
SFDX - asking for bulk status for id 7516E000002DckQQAS, jobId 7506E000002QQ3zQAG
SFDX - received bulk status for id 7516E000002DckQQAS, jobId 7506E000002QQ3zQAG - state: Completed
Issued UPSERT bulk request to object (MarketRelation__c) - id 7516E000002DckkQAC, jobId 7506E000002QQ4JQAW - state: Queued
SFDX - asking for bulk status for id 7516E000002DckkQAC, jobId 7506E000002QQ4JQAW
SFDX - received bulk status for id 7516E000002DckkQAC, jobId 7506E000002QQ4JQAW - state: Completed
Issued UPSERT bulk request to object (Consent__c) - id 7516E000002DckuQAC, jobId 7506E000002QQ4OQAW - state: Queued
SFDX - asking for bulk status for id 7516E000002DckuQAC, jobId 7506E000002QQ4OQAW
SFDX - received bulk status for id 7516E000002DckuQAC, jobId 7506E000002QQ4OQAW - state: Completed
Finished upsert of data

Once I’m done testing a particular data set I can use the –delete-accounts flag to my script to delete data using the Bulk API as well. Here I actually combined force:data:soql:query and force:data:bulk:delete to first retrieve the ID’s of the records I need to delete and then kick off the required Bulk API delete requests. Again easy peasy. And repeatable…

$ ./upsert mheisterberg@example.com.appdev --delete-accounts
SFDX - Org for mheisterberg@example.com.appdev is connected...

Are you sure?
 ▸ No
 Yes

Received 32463 records
Issued DELETE bulk request to object (Account) - id 7516E000002DckLQAS, jobId 7506E000002QQ3uQAG - state: Queued
SFDX - asking for bulk status for id 7516E000002DckLQAS, jobId 7506E000002QQ3uQAG
SFDX - received bulk status for id 7516E000002DckLQAS, jobId 7506E000002QQ3uQAG - state: InProgress
SFDX - asking for bulk status for id 7516E000002DckLQAS, jobId 7506E000002QQ3uQAG
SFDX - received bulk status for id 7516E000002DckLQAS, jobId 7506E000002QQ3uQAG - state: Completed
Performed delete...

Only issue I had here really was that node.js has a maximum buffer size of 200kb to stdin so I could not simply read the stdin response from the SOQL query as it may be pretty big. Instead I pipe to a tmp-file and read that back in and parse as JSON. Not ideal but it gets the job done.

The code of the script itself is for the customers eyes only but the source for the helpers is available as sfdx-bulk-helper on Github and sfdx-bulk-helper on npm.

YMMV!

 

Lightning Logger

In my Lightning components I always find logging and remembering how to navigate to urls using events to be an issue so I wrote this small utility base component that I then extend when creating new components. The base component share its helper with the subcomponents which allows for easy reuse of the utility functionality. The utility code provides both logging and various other utility functions such as navigating to other objects, presenting toasts etc.

Component definition of the base component is simple (note the {!v.body} which is key for abstract components to make sure the markup of child components appear):

<aura:component abstract="true" extensible="true">
 <aura:handler name="init" value="{!this}" action="{!c.doinit}" />
 {!v.body}
</aura:component>

Controller is likewise simple with basically only a callout to the helper to initialize a named Logger instance and store it in a variable called logger in the helper.

({
  doinit: function(component, event, helper) {
    const logger = helper.getLogger('SFLC_LightningHelper');
    logger.trace('Initializing LightningHelper');
    helper.logger = logger;
  }
})

To use it from another component you first extend the base component:

<aura:component implements="flexipage:availableForAllPageTypes,force:hasRecordId" extends="c:BaseComponent" controller="Foo_LCC">
  <aura:attribute name="recordId" type="Id" />
  <aura:handler name="init" value="{!this}" action="{!c.doinit}" />
  ...
  ...
</aura:component>

Then in the component controller initialization event create a named utility object (actually names the logger) and store it the “util” variable in the helper making it accessible using helper.util and the logger using helper.util.logger.

({
  doinit: function(component, event, helper) {
    // build utility
    const utility = helper.buildUtilityObject('MyComponent');
    helper.util = utility;
    
    // load data
    helper.loadData(component);
  }
})

From here on out you can simply use helper.util.logger.info, helper.util.logger.debug etc. to log for the component. All log messages are output using the named logger. The log level  which by default is NONE (meaning no output is output) is controllable using a URL parameter and loggers may be turned on and off using a URL parameter as well. Please note that the URL format used here doesn’t live up to the changes coming to Lightning URL’s in Summer ’18 or Winter ’18 (cannot remember which release).

Using the utility functionality can be done from controller and helper as shown in the loadData method using both utility functionality to invoke a remote action and to log:

({
  loadData: function(component) {
    // load data
    const helper = this;
    helper.util.invokeRemoteAction(component, 
      'getData', {'recordId': component.get("v.recordId")}, 
      function(err, data) {
        if (err) {
          helper.util.toast.error('Data Load Error', 'Unable to load data from server ('+ err +'). If the issue persists contact your System Administrator.', {sticky: true}); 
          return;
        }
        helper.util.log.debug('Received data from endpoint', data); 
      }
    )
  }
})

Zip file with Controller and Helper: BaseComponent

Salesforce Release Notes

So I’m reading through the Salesforce Spring 18 release notes for a customer project starting later this week for specifics on changes related to GDPR. So happy for searching as the release notes are a wopping 459 pages. What! Seriously? Oh yes…

The way I tend to go through the release notes is to get the PDF version as I find it easier to skim and search through. The release notes are also available in HTML which can be good if you know you’re only looking for say Lightning related changes or AppDev related changes.

Only other piece of advise – make sure you’re sitting comfortably and have coffee at the ready!

Salesforce Spring 18 release notes (PDF)
Salesforce Spring 18 release notes (HTML)

Beta of new Lightning Component Library included in Winter ’18

Was pleasantly surprised to see the beta of the new Lightning Component library being included in the Winter ’18 release. Quoting from the release notes (see page 533):


Find components more easily with the searchable component library. Preview the look and feel of components with interactive examples. To explore the new component library, go to https://<myDomain&gt;.lightning.force.com/componentReference/suite.app where myDomain is the name of your custom Salesforce domain. You can also continue to use /auradocs/reference.app in your org. And the Component Reference section in the Lightning Components Developer Guide continues to be available to you. …snip… Let’s look at some of the highlights. Gone are the days when you had to imagine how a Lightning component renders and behaves in your browser. Many components now have interactive examples and source code for common patterns. You can also learn about how a component is used through a list of examples.

The component library is really for the built in base components what the auradocs at /auradocs/reference.app is for your custom components and with the additional of all the new base components in Winter ’18 this is a nice addition.

The best way to learn a platform is to use a platform

Wow what a week it’s been. First week back from vacation and I’m diving right into a sprint of stuff that needs to be delivered to the customer. My task for the week has been develop a connectivity layer between Salesforce and Dropbox using OAuth. This task has taken me on quite a learning journey.

Now I’ll call myself quite a seasoned programmer but ever since joining Salesforce 9 months ago I’ve had to relearn a lot of stuff. A new programming language in Apex, new framework technologies such as Salesforce Lightning Design System (SLDS) and the Lightning Platform and the entire underlying Salesforce Platform which is enormous. There are just different ways to do stuff… Previously I would have had my choice of any language, technology and framework to solve the issue but now those are kind of given.

Just this afternoon as I was putting the final semi-colons in the core OAuth layer (Apex), the Dropbox specific OAuth layer (Apex), the business specific facade class in front of those layers (Apex) and the management pages for the integration (Lightning) I was reminded of an old quote from an old biochemistry text book of mine: “The best way to learn a subject is the teach the subject”. And that continues to hold true and is equally true when saying “the best way to learn a platform is to build on the platform”. I’ve learned much stuff about Apex and Lightning in the past week. All the cool stuff you can do but also some of the crazy stuff that falls into that “you have have to know that”-category. But for all those things it holds true that you have to spend the time to reap the benefits.

For now I’ll just say that although the Apex language has it quirks and the compiler is definitely showing age (and Salesforce knows this – a new compiler is being developed or at least that’s what I heard in a Dreamforce 2016 session) there is still much so much power in the language. Is there stuff you cannot do? Sure! But there are cool interfaces like System.StubProvider which I read and hear little talk about. The built in support to unit tests and mocking of HTTP requests is awesome and allows you to quickly and easily stub out calls to external services. The runtime screams with performance and I’m pretty impressed about the speed with which my code runs.

Good stuff!

So in closing – I have my OAuth layer done, unit tested and commited. I’ll surely be blogging about how to chose to build it so others may learn from it if they want to spend the time and learn the platform.

Developing Salesforce Lightning Components that are visible at design time but not at runtime

So this can clearly be labelled as a “Lightning Lesson from the Field”. As you start to develop more complicated Salesforce Lightning applications – and why wouldn’t you – you as I have done start seeing great power in hidden components. By hidden components I mean components that contribute code or does “something” but which does not have a UI. Thes are very easy to do but have a big drawback as they are also invisible at design time making them near impossible to find the Lightning AppBuilder. To work around this I’ve come up with a hack that has proven itself very useful.

By using a combination of two simple attributes and 3 lines of JavaScript I can make the component markup visible at design time but invisible at runtime. Or if I need to toggle it on for runtime as well using an attribute I can set from the Lightning AppBuilder.

The video below goes more in depth and illustrates the concept. Happy coding.

Deploy your own Salesforce Workbench on Heroku at the click of a button

The other day Salesforce Workbench was having issues. Generally it kept returning errors and SOQL queries took forever and timed out. Now Salesforce Workbech is a LAMP app that runs on Heroku and it turns out it is actually possible to deploy your own instance on Heroku using a simple Heroku Button. To do this simply follow the below steps (you need to have an account but if you don’t simply sign up):

  1. Go to the project page at https://elements.heroku.com/buttons/jdrishe/salesforce-workbench
  2. Click the Deploy to Heroku button (use the button on the bottom and not the one on the top right as the one on the top right deploys and older version)
  3. Log into the deployed app (remember to add your security token after the password but I guess you knew that!!) and you are done!

Now you have your own instance of Salesforce Workbench running on Heroku. So nice. Below is a small video I recorded showing the actual steps and realtime – just over a minute and you have your own instance! Boom!!