- 11412 Views
- 7 replies
- 4 kudos
Resolved! Retrieve job id and run id from scala
I need to retrieve job id and run id of the job from a jar file in Scala.When I try to compile below code in IntelliJ, below error is shown.import com.databricks.dbutils_v1.DBUtilsHolder.dbutils object MainSNL { @throws(classOf[Exception]) de...
- 11412 Views
- 7 replies
- 4 kudos
- 4 kudos
Maybe its worth going through the Task Parameter variables section of the below dochttps://docs.databricks.com/data-engineering/jobs/jobs.html#task-parameter-variables
- 4 kudos
- 6850 Views
- 1 replies
- 2 kudos
Resolved! Databricks jobs create API throws unexpected error
Databricks jobs create API throws unexpected errorError response :{"error_code": "INVALID_PARAMETER_VALUE","message": "Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size"}Any idea on this?
- 6850 Views
- 1 replies
- 2 kudos
- 2 kudos
Could you please specify num_workers in the json body and try API again.Also, another recommendation can be configuring what you want in UI, and then pressing “JSON” button that should show corresponding JSON which you can use for API
- 2 kudos
- 2155 Views
- 1 replies
- 1 kudos
Correlated Column Exception in Spark SQL
Hi Johan,Were you able to resolve the correlated column exception issue? I have been stuck on this since past week. If you can guide me that will be alot of help.Thanks.
- 2155 Views
- 1 replies
- 1 kudos
- 1 kudos
Seems to be a duplicate of your comment on https://community.databricks.com/s/question/0D53f00001XCuCACA1/correlated-column-exception-in-sql-udf-when-using-udf-parameters. I guess you did that to be able to put other tags?
- 1 kudos
- 31930 Views
- 13 replies
- 12 kudos
Resolved! Is there a way to run notebooks concurrently in same session?
tried using-dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)but it takes 20 seconds to start new session. %run uses same session but cannot figure out how to use it to run notebooks concurrently.
- 31930 Views
- 13 replies
- 12 kudos
- 12 kudos
I’ve been struggling with opening multiple browser sessions to open more than one notebook at a time.
- 12 kudos
- 1793 Views
- 0 replies
- 20 kudos
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endp...
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endpoint via API/SDK
- 1793 Views
- 0 replies
- 20 kudos
- 16732 Views
- 5 replies
- 8 kudos
Resolved! How to create delta table with identity column.
I'm sure this is probably some oversight on my part, but I don't see it. I'm trying to create a delta table with an identity column. I've tried every combination of the syntax I can think of. %sql create or replace table IDS.picklist ( picklist_id...
- 16732 Views
- 5 replies
- 8 kudos
- 8 kudos
I was also having this same error and my cluster was running Databricks Runtime Version 9.1 so I changed it to 11.0 and it worked.
- 8 kudos
- 2220 Views
- 0 replies
- 0 kudos
Why use databricks over other tools ?
What is something special about databricks.What databricks provides that no other tool in the market provides ?How can I convince some other person to use databricks and not some other tool ?
- 2220 Views
- 0 replies
- 0 kudos
- 1567 Views
- 0 replies
- 0 kudos
Big data ingest into Delta Lake
I have a feature table in BQ that I want to ingest into Delta Lake. This feature table in BQ has 100TB of data. This table can be partitioned by DATE.What best practices and approaches can I take to ingest this 100TB? In particular, what can I do to ...
- 1567 Views
- 0 replies
- 0 kudos
- 3588 Views
- 2 replies
- 4 kudos
DLT schema ambiguity
I have schema:| |-- costCentres: struct (nullable = true) | | |-- dimension1: struct (nullable = true) | | | |-- name: string (nullable = true) | | | |-- value: string (nullable = true) | | |-- dimension10: struct...
- 3588 Views
- 2 replies
- 4 kudos
- 4 kudos
I've got the same problem - but using a SQL Select statement (with some explodes).
- 4 kudos
- 2379 Views
- 1 replies
- 1 kudos
Resolved! How do analytics access the lake house
- 2379 Views
- 1 replies
- 1 kudos
- 1 kudos
Through mounting your Databricks resource and your storage resource. Here is a sample using azure blob storage Azure Blob storage - Azure Databricks | Microsoft Docs
- 1 kudos
- 2446 Views
- 1 replies
- 0 kudos
‘How do you get cost of a notebook run?’
‘How do you get cost of a notebook run?’
- 2446 Views
- 1 replies
- 0 kudos
- 0 kudos
You can check your cloud provider's portal. Go to the subscription > costs field and you should be able to see the costs of the VMs and Databricks. For more granular information, consider installing overwatch.Environment Setup :: Overwatch (databrick...
- 0 kudos
- 2334 Views
- 1 replies
- 1 kudos
- 2334 Views
- 1 replies
- 1 kudos
- 1 kudos
Table ACLs, Hashing, Anonymization and Pseudonymization of PII to name a few.You can learn everything in the databricks academy course for professional data engineering.
- 1 kudos
- 5925 Views
- 2 replies
- 5 kudos
Resolved! what are the key Data engineering problems that databricks solve?
what are the problem that databricks address from typical data engineering prespective and comparing with other cloud DE tools.
- 5925 Views
- 2 replies
- 5 kudos
- 5 kudos
Annoying things databricks solvesSane Data Movement (Fast Parallelized Compute, Table Versioning and History)Environment Management (spark + delta + java) are installed out-of-the-boxCost and Job Monitoring (Overwatch)I've only worked with it for 6 m...
- 5 kudos
- 1599 Views
- 1 replies
- 1 kudos
PII restriction
Can you enforce PII export restrictions at the user level?
- 1599 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes you can, by enforcing table ACLs (premium plan feature). A rule of thumb though is to do this on a group level.Table access control | Databricks on AWS
- 1 kudos
- 3834 Views
- 3 replies
- 1 kudos
Resolved! Is there a way to protect the secrets on databricks?
Hi all, I’ve a lot of secrets on databricks, however my users are able to see it when they make a simple loop over the secret.is there a way to prevent that?regards
- 3834 Views
- 3 replies
- 1 kudos
- 1 kudos
Do use the Databricks CLI or API 2.0 to manage secrets. Don't leave them in your notebooks for everyone to see, same applies for salting or hash strings.Secret management - Azure Databricks | Microsoft Docs
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
