- 3817 Views
- 1 replies
- 0 kudos
AWS Databricks VS AWS EMR
HiWhich services should I use for data lake implementation?any cost comparison between Databricks and aws emr.which one is best to choose
- 3817 Views
- 1 replies
- 0 kudos
- 0 kudos
@AH that depends on use case, if your implementation involves Data Lake, ML, Data engineering tasks better to go with databricks as it has got good UI and there good governance using unity catalog for your data lake and you have good consumer tool su...
- 0 kudos
- 4110 Views
- 1 replies
- 1 kudos
Resolved! System billing usage table - Usage column
Hello experts,Could someone please explain what is exactly contained into the column usage in the system.billing.usage table?We ran specific queries in a cluster trying to calculate the cost and we observe that the DBUs shown in the system table are ...
- 4110 Views
- 1 replies
- 1 kudos
- 1 kudos
@elgeo both should be same, untill if somehow we miss to pick proper plan DBU price, usage column will have complete information related to sku name and DBU units etc... if you use azure databricks calculator and compare we should see similar result
- 1 kudos
- 1845 Views
- 0 replies
- 0 kudos
Suggestions for python its not working
Tab and Shift tab its not suggestin nothing in python codeTab only work as indentation and shift + tab do nothing.What can I do?
- 1845 Views
- 0 replies
- 0 kudos
- 8551 Views
- 0 replies
- 0 kudos
How to retrieve a Job Name from the SparkContext
We are currently starting to build certain data pipelines using Databricks.For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the ...
- 8551 Views
- 0 replies
- 0 kudos
- 5602 Views
- 4 replies
- 2 kudos
Roadmap on export menu option for SQL Query and Dashboard Types in Workspace
Are there plans for an export option for SQL Query and SQL Dashboard in the Workspace explorer screen similar to notebooks?Background: Need a way to export and backup any queries and dashboards to save design work and move from staging environments ...
- 5602 Views
- 4 replies
- 2 kudos
- 2 kudos
Th best option would be to have them just under git Repo (especially dashboards).
- 2 kudos
- 8519 Views
- 6 replies
- 1 kudos
Long run time with %run command
My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...
- 8519 Views
- 6 replies
- 1 kudos
- 1574 Views
- 0 replies
- 0 kudos
case statements return same value
I have these 4 case statements count(*) as Total_claim_reciepts,count(case when claim_id like '%M%' and receipt_flag = 1 and is_firstpassclaim = 1 then 0 else claim_id end) as Total_claim_reciepts,count(case when claim_status ='DENIED' and claim_repa...
- 1574 Views
- 0 replies
- 0 kudos
- 9653 Views
- 2 replies
- 0 kudos
NATIVE_XML_DATA_SOURCE_NOT_ENABLED
I'm trying to read an xml file and receiving the following error. I've installed the maven library spark xml to the cluster, however I'm receiving the error. is there anything i'm missing?ErrorAnalysisException: [NATIVE_XML_DATA_SOURCE_NOT_ENABLED] N...
- 9653 Views
- 2 replies
- 0 kudos
- 0 kudos
i've tried already spark.read.format('com.databricks.spark.xml'). it receives the same error.
- 0 kudos
- 2365 Views
- 0 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam, saying I have exceeded ...
- 2365 Views
- 0 replies
- 0 kudos
- 2336 Views
- 1 replies
- 0 kudos
how to save variables in one notebook to be imported into another?
Say, I have a list of values, dictionaries, variable names in `notebook1.ipynb` that I'd like to re-use / import in another `notebook2.ipynb`. For example, in `notebook1.ipynb`, I have the following: var1 = "dallas" var_lst = [100, 200, 300, 400, ...
- 2336 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use %run ./notebook2 after defining variables in notebook1So notebook2 will use the variables defined in notebook1
- 0 kudos
- 4070 Views
- 1 replies
- 0 kudos
How to save the best model checkpoi through the epochs of a deep learning network through callbacks?
I have create a neural network and I am training the model with the code as below. The code fails to write to the databricks file storage. is there any other way to write the checkpoint to databricks storage or to an s3 bucket directly?custom_early_...
- 4070 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod ,I am not able to save it to local storage in databricks dbfs also. It is showing invalid operation when I am trying to save to databricks file storage. Additionally, I have valid aws credentials with which I am able to save a model t...
- 0 kudos
- 2246 Views
- 0 replies
- 0 kudos
Do we need to Request Databrikcs to Enable MOSIAC ML
HI Team,I am not seeing any specific articles/guides to use MOSIAC ML on Databricks. After Acquiring MOSIAC ML does anything got changed in terms of MOSIAC ML Use or just use just regular function
- 2246 Views
- 0 replies
- 0 kudos
- 3197 Views
- 3 replies
- 0 kudos
Databricks Delta table Insert Data Error
When trying to insert data into the Delta table in databricks, an error occurs as shown below. [TASK_WRITE_FAILED] Task failed while writing rows to abfss://cont-01@dlsgolfzon001.dfs.core.windows.net/dir-db999_test/D_RGN_INFO_TMP.In SQL, the results ...
- 3197 Views
- 3 replies
- 0 kudos
- 0 kudos
seems ok to me, have you tried to display the data from table A and also the B/C join?
- 0 kudos
- 3318 Views
- 1 replies
- 0 kudos
Retrieve DBU per query executed
Hello experts,Do you know how we can retrieve the DBUs consumed for a specific query?Thank you
- 3318 Views
- 1 replies
- 0 kudos
- 0 kudos
I couldn't find a metadata table. However the workaround is to multiply the DBU of the current cluster (retrieve it either online or to be more accurate from the compute page at the right) and multiply it with the time in minutes that the query took ...
- 0 kudos
- 1785 Views
- 1 replies
- 0 kudos
how to make distributed predictions with sklearn model?
So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...
- 1785 Views
- 1 replies
- 0 kudos
