- 23576 Views
- 13 replies
- 12 kudos
Resolved! Is there a way to run notebooks concurrently in same session?
tried using-dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)but it takes 20 seconds to start new session. %run uses same session but cannot figure out how to use it to run notebooks concurrently.
- 23576 Views
- 13 replies
- 12 kudos
- 12 kudos
I’ve been struggling with opening multiple browser sessions to open more than one notebook at a time.
- 12 kudos
- 1137 Views
- 0 replies
- 20 kudos
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endp...
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endpoint via API/SDK
- 1137 Views
- 0 replies
- 20 kudos
- 12788 Views
- 5 replies
- 8 kudos
Resolved! How to create delta table with identity column.
I'm sure this is probably some oversight on my part, but I don't see it. I'm trying to create a delta table with an identity column. I've tried every combination of the syntax I can think of. %sql create or replace table IDS.picklist ( picklist_id...
- 12788 Views
- 5 replies
- 8 kudos
- 8 kudos
I was also having this same error and my cluster was running Databricks Runtime Version 9.1 so I changed it to 11.0 and it worked.
- 8 kudos
- 1652 Views
- 0 replies
- 0 kudos
Why use databricks over other tools ?
What is something special about databricks.What databricks provides that no other tool in the market provides ?How can I convince some other person to use databricks and not some other tool ?
- 1652 Views
- 0 replies
- 0 kudos
- 1046 Views
- 0 replies
- 0 kudos
Big data ingest into Delta Lake
I have a feature table in BQ that I want to ingest into Delta Lake. This feature table in BQ has 100TB of data. This table can be partitioned by DATE.What best practices and approaches can I take to ingest this 100TB? In particular, what can I do to ...
- 1046 Views
- 0 replies
- 0 kudos
- 2399 Views
- 2 replies
- 4 kudos
DLT schema ambiguity
I have schema:| |-- costCentres: struct (nullable = true) | | |-- dimension1: struct (nullable = true) | | | |-- name: string (nullable = true) | | | |-- value: string (nullable = true) | | |-- dimension10: struct...
- 2399 Views
- 2 replies
- 4 kudos
- 4 kudos
I've got the same problem - but using a SQL Select statement (with some explodes).
- 4 kudos
- 1653 Views
- 1 replies
- 1 kudos
Resolved! How do analytics access the lake house
- 1653 Views
- 1 replies
- 1 kudos
- 1 kudos
Through mounting your Databricks resource and your storage resource. Here is a sample using azure blob storage Azure Blob storage - Azure Databricks | Microsoft Docs
- 1 kudos
- 1583 Views
- 1 replies
- 0 kudos
‘How do you get cost of a notebook run?’
‘How do you get cost of a notebook run?’
- 1583 Views
- 1 replies
- 0 kudos
- 0 kudos
You can check your cloud provider's portal. Go to the subscription > costs field and you should be able to see the costs of the VMs and Databricks. For more granular information, consider installing overwatch.Environment Setup :: Overwatch (databrick...
- 0 kudos
- 1543 Views
- 1 replies
- 1 kudos
- 1543 Views
- 1 replies
- 1 kudos
- 1 kudos
Table ACLs, Hashing, Anonymization and Pseudonymization of PII to name a few.You can learn everything in the databricks academy course for professional data engineering.
- 1 kudos
- 4193 Views
- 2 replies
- 5 kudos
Resolved! what are the key Data engineering problems that databricks solve?
what are the problem that databricks address from typical data engineering prespective and comparing with other cloud DE tools.
- 4193 Views
- 2 replies
- 5 kudos
- 5 kudos
Annoying things databricks solvesSane Data Movement (Fast Parallelized Compute, Table Versioning and History)Environment Management (spark + delta + java) are installed out-of-the-boxCost and Job Monitoring (Overwatch)I've only worked with it for 6 m...
- 5 kudos
- 1025 Views
- 1 replies
- 1 kudos
PII restriction
Can you enforce PII export restrictions at the user level?
- 1025 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes you can, by enforcing table ACLs (premium plan feature). A rule of thumb though is to do this on a group level.Table access control | Databricks on AWS
- 1 kudos
- 2400 Views
- 3 replies
- 1 kudos
Resolved! Is there a way to protect the secrets on databricks?
Hi all, I’ve a lot of secrets on databricks, however my users are able to see it when they make a simple loop over the secret.is there a way to prevent that?regards
- 2400 Views
- 3 replies
- 1 kudos
- 1 kudos
Do use the Databricks CLI or API 2.0 to manage secrets. Don't leave them in your notebooks for everyone to see, same applies for salting or hash strings.Secret management - Azure Databricks | Microsoft Docs
- 1 kudos
- 18933 Views
- 12 replies
- 11 kudos
Resolved! Is there any difference between performance of Python and SQL ?
I read somewhere that Python code is converted to SQL at the end. So is it true or there is any difference in performance while working with Scala, Python or SQL ?
- 18933 Views
- 12 replies
- 11 kudos
- 11 kudos
To add on the consideration of UDFs, try to consider using HOFs (Higher Order Functions) whenever possible first as there is a signifcant performance benefit as seen here.
- 11 kudos
- 1090 Views
- 0 replies
- 1 kudos
Structured streaming vs Confluent Kstream
For Ultra low latency customer facing App, I am curious on cost efficiency between Structured streaming and Kstream; which work better in terms of cost ? Though still achieving the ultra low latency and quality outcome. Appreciate any thoughts from p...
- 1090 Views
- 0 replies
- 1 kudos
- 1269 Views
- 1 replies
- 1 kudos
Resolved! How to improve the performance of small delta tables?
How to improve the performance of small delta tables?
- 1269 Views
- 1 replies
- 1 kudos
- 1 kudos
Use databricks optimize command on delta tables. It will regroup al the files and provides better performance​
- 1 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
