- 1910 Views
- 1 replies
- 0 kudos
Read arguments in a scala note invoked by a job.
In a scala note, how to I read input arguments (e.g. those proved by a job that runs a scala notebook). In python, dbutils.notebook.entry_point.getCurrentBindings() works. How about for scala.
- 1910 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Robert Russell​ You can use dbutils.notebook.getContext.currentRunId in scala notebooks. Other methods are also available likedbutils.notebook.getContext.jobGroupdbutils.notebook.getContext.rootRunId dbutils.notebook.getContext.tags etc...You ...
- 0 kudos
- 838 Views
- 0 replies
- 0 kudos
Hi FOLKS, could you please guide me, why my points not reflecting in reward store, it is showing 0.
Hi FOLKS, could you please guide me, why my points not reflecting in reward store, it is showing 0.
- 838 Views
- 0 replies
- 0 kudos
- 4724 Views
- 2 replies
- 0 kudos
Reading Password protected excel(.xlsx) file in databricks
I want to read password protected excel file and load the data delta table.Can you pleas let me know how this can be achieved in databricks?
- 4724 Views
- 2 replies
- 0 kudos
- 0 kudos
df = spark.read.format("com.crealytics.spark.excel")\ .option("dataAddress", "'Base'!A1")\ .option("header", "true")\ .option("workbookPassword", "test")\ .load("test.xlsx")display(df)
- 0 kudos
- 3290 Views
- 2 replies
- 2 kudos
- 3290 Views
- 2 replies
- 2 kudos
- 2 kudos
As @Werner Stinckens​ said, it would be ok. But generally decimal column joins are not recommended as other factors come into play like the precision, length etc...Also when you are joining in on decimal columns, be sure to check out the abs value of...
- 2 kudos
- 3081 Views
- 1 replies
- 1 kudos
Does CACHE TABLE/VIEW have a create or replace like view?
I'm trying to cache data/queries that we normally have as temporary views that get replaced when the code is run based on dynamic python. What I'd like to know is will CACHE TABLE get overwritten each time you run it? Is it smart enough to recognize ...
- 3081 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Matt Fury​ Yes...I guess cache overwrites each time you run it because for me it took nearly same amount of time for 1million records to be cached. However, you can check whether the table is cached or not using .storageLevel method. E.g. I have...
- 1 kudos
- 7159 Views
- 4 replies
- 4 kudos
Resolved! Azure Databricks Default DNS
My Azure Databricks workspace default DNS is #168.63.129.16, this DNS doesn't seem to resolve azure storage accounts which were created a year ago, after tweaking the cluster to use 8.8.8.8 then able to resolve desired storage accounts, is there a d...
- 7159 Views
- 4 replies
- 4 kudos
- 4 kudos
IP address 168.63.129.16 is a virtual public IP address that is used to facilitate a communication channel to Azure platform resources. Customers can define any address space for their private virtual network in Azure. Therefore, the Azure platform...
- 4 kudos
- 3486 Views
- 3 replies
- 3 kudos
"No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas
My Mongo Atlas connect url is like mongodb+srv://<srv_hostname>I don't want to use direct url like mongodb://<hostname1, hostname2, hostname3....> because our Mongo Atlas global clusters have many hosts. It would be hard to maintain.Our java programs...
- 3486 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Raymond Lai​ The issue looks to be on the Mongo DB connector. The connection is created and maintained by the mongo-spark connector. You can try using the direct mongodb hosts in the connection string instead of SRV to avoid doing DNS lookups or...
- 3 kudos
- 8563 Views
- 4 replies
- 5 kudos
Is it reasonable for the process "Determining the location of DBIO file fragments." to take me 7 hours?
I only have 1000 columns. Each column has 252 rows, so there are only 252000 data points.How come it can route tasks for the best-cached locality for 7 hours?
- 8563 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Cheuk Hin Christophe Poon​ have you optimize your table anytime since it's creation? If not, then optimize may take some time depending on the no of underlying files.Please try to run optimize manually as described in below document:https://docs....
- 5 kudos
- 6031 Views
- 4 replies
- 4 kudos
How to use delta live table with google cloud storage
Hi Team I have been working on a POC exploring delta live table with GCS location. I have some doubts :how to access the gcs bucket. We have connection established using databricks service account. In a normal cluster creation , we go to cluster page...
- 6031 Views
- 4 replies
- 4 kudos
- 4 kudos
Kindly mount the DBFS location to GCS cloud storage, see belowMounting cloud object storage on Databricks | Databricks on Google Cloud
- 4 kudos
- 9973 Views
- 4 replies
- 3 kudos
Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.
Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.
- 9973 Views
- 4 replies
- 3 kudos
- 3 kudos
Adding some more points to @karthik p​ 's answer.Use kryo serializer instead of java serializer.Use an optimised garbage collector such as G1GC.Use partitioning wisely on a field.
- 3 kudos
- 3935 Views
- 3 replies
- 7 kudos
Databricks Log4J Custom Appender Not Working as expected
I'm trying to figure out how a custom appender should be configured in a Databricks environment but I cannot figure it out.When cluster is running, in `driver logs`, time is displayed as 'unknown' for my custom log file and when cluster is stopped, c...
- 3935 Views
- 3 replies
- 7 kudos
- 7 kudos
We're having the same problem with 11.3 LTS. Are there any updates? We would like to deliver log4j messages from Databricks Notebooks to custom log files and then upload those to S3 or DBFS. Best
- 7 kudos
- 47117 Views
- 3 replies
- 10 kudos
Resolved! How to get all occurrences of duplicate records in a PySpark DataFrame based on specific columns?
Hi,I need to find all occurrences of duplicate records in a PySpark DataFrame. Following is the sample dataset:# Prepare Data data = [("A", "A", 1), \ ("A", "A", 2), \ ("A", "A", 3), \ ("A", "B", 4), \ ("A", "B", 5), \ ("A", "C", ...
- 47117 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi,​In my experience, if you use dropDuplicates(), Spark will keep a random row.​Therefore, you should define a logic to remove duplicated rows.
- 10 kudos
- 8873 Views
- 5 replies
- 19 kudos
Practice Exams for Databricks Certified Data Engineer Professional exam
Can anyone help with official Practice Exams set for Databricks Certified Data Engineer Professional exam, like we have below for Databricks Certified Data Engineer AssociatePractice exam for the Databricks Certified Data Engineer Associate exam
- 8873 Views
- 5 replies
- 19 kudos
- 19 kudos
hi @Shalabh Agarwal​ I am not able to find any official practice paper. it is still not available.
- 19 kudos
- 3840 Views
- 1 replies
- 2 kudos
External APIs
Does databricks provide a way to integrate to external sw/API's? Whether it is in the form of UDF/external function? Can somebody point me how this can be achieved? My use case is to talk to external API's from databricks to perform certain operation...
- 3840 Views
- 1 replies
- 2 kudos
- 2 kudos
You can write your own code to fetch data from external API.Example: https://insightsndata.com/how-to-call-rest-api-store-data-in-databricks-8383f2458d7d
- 2 kudos
- 4772 Views
- 5 replies
- 0 kudos
Resolved! Is there a way to get cdc data from salesforce to databricks? Can a smart pipeline be built to get near real time data from salesforce into delta lake?
Currently, we have daily batch running to extract data from salesforce into csv file (adls) which is further copied to delta tables for transformation. We are now looking to implement a solution which can extract real-time data changes on salesforce ...
- 4772 Views
- 5 replies
- 0 kudos
- 0 kudos
On Azure you can try using SAP CDC connector for Data Factory:https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-introduction-architecture
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
