- 1723 Views
- 1 replies
- 1 kudos
Correlated Column Exception in Spark SQL
Hi Johan,Were you able to resolve the correlated column exception issue? I have been stuck on this since past week. If you can guide me that will be alot of help.Thanks.
- 1723 Views
- 1 replies
- 1 kudos
- 1 kudos
Seems to be a duplicate of your comment on https://community.databricks.com/s/question/0D53f00001XCuCACA1/correlated-column-exception-in-sql-udf-when-using-udf-parameters. I guess you did that to be able to put other tags?
- 1 kudos
- 26384 Views
- 13 replies
- 12 kudos
Resolved! Is there a way to run notebooks concurrently in same session?
tried using-dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)but it takes 20 seconds to start new session. %run uses same session but cannot figure out how to use it to run notebooks concurrently.
- 26384 Views
- 13 replies
- 12 kudos
- 12 kudos
I’ve been struggling with opening multiple browser sessions to open more than one notebook at a time.
- 12 kudos
- 1345 Views
- 0 replies
- 20 kudos
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endp...
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endpoint via API/SDK
- 1345 Views
- 0 replies
- 20 kudos
- 14071 Views
- 5 replies
- 8 kudos
Resolved! How to create delta table with identity column.
I'm sure this is probably some oversight on my part, but I don't see it. I'm trying to create a delta table with an identity column. I've tried every combination of the syntax I can think of. %sql create or replace table IDS.picklist ( picklist_id...
- 14071 Views
- 5 replies
- 8 kudos
- 8 kudos
I was also having this same error and my cluster was running Databricks Runtime Version 9.1 so I changed it to 11.0 and it worked.
- 8 kudos
- 1832 Views
- 0 replies
- 0 kudos
Why use databricks over other tools ?
What is something special about databricks.What databricks provides that no other tool in the market provides ?How can I convince some other person to use databricks and not some other tool ?
- 1832 Views
- 0 replies
- 0 kudos
- 1209 Views
- 0 replies
- 0 kudos
Big data ingest into Delta Lake
I have a feature table in BQ that I want to ingest into Delta Lake. This feature table in BQ has 100TB of data. This table can be partitioned by DATE.What best practices and approaches can I take to ingest this 100TB? In particular, what can I do to ...
- 1209 Views
- 0 replies
- 0 kudos
- 2829 Views
- 2 replies
- 4 kudos
DLT schema ambiguity
I have schema:| |-- costCentres: struct (nullable = true) | | |-- dimension1: struct (nullable = true) | | | |-- name: string (nullable = true) | | | |-- value: string (nullable = true) | | |-- dimension10: struct...
- 2829 Views
- 2 replies
- 4 kudos
- 4 kudos
I've got the same problem - but using a SQL Select statement (with some explodes).
- 4 kudos
- 1867 Views
- 1 replies
- 1 kudos
Resolved! How do analytics access the lake house
- 1867 Views
- 1 replies
- 1 kudos
- 1 kudos
Through mounting your Databricks resource and your storage resource. Here is a sample using azure blob storage Azure Blob storage - Azure Databricks | Microsoft Docs
- 1 kudos
- 1891 Views
- 1 replies
- 0 kudos
‘How do you get cost of a notebook run?’
‘How do you get cost of a notebook run?’
- 1891 Views
- 1 replies
- 0 kudos
- 0 kudos
You can check your cloud provider's portal. Go to the subscription > costs field and you should be able to see the costs of the VMs and Databricks. For more granular information, consider installing overwatch.Environment Setup :: Overwatch (databrick...
- 0 kudos
- 1774 Views
- 1 replies
- 1 kudos
- 1774 Views
- 1 replies
- 1 kudos
- 1 kudos
Table ACLs, Hashing, Anonymization and Pseudonymization of PII to name a few.You can learn everything in the databricks academy course for professional data engineering.
- 1 kudos
- 4743 Views
- 2 replies
- 5 kudos
Resolved! what are the key Data engineering problems that databricks solve?
what are the problem that databricks address from typical data engineering prespective and comparing with other cloud DE tools.
- 4743 Views
- 2 replies
- 5 kudos
- 5 kudos
Annoying things databricks solvesSane Data Movement (Fast Parallelized Compute, Table Versioning and History)Environment Management (spark + delta + java) are installed out-of-the-boxCost and Job Monitoring (Overwatch)I've only worked with it for 6 m...
- 5 kudos
- 1184 Views
- 1 replies
- 1 kudos
PII restriction
Can you enforce PII export restrictions at the user level?
- 1184 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes you can, by enforcing table ACLs (premium plan feature). A rule of thumb though is to do this on a group level.Table access control | Databricks on AWS
- 1 kudos
- 2777 Views
- 3 replies
- 1 kudos
Resolved! Is there a way to protect the secrets on databricks?
Hi all, I’ve a lot of secrets on databricks, however my users are able to see it when they make a simple loop over the secret.is there a way to prevent that?regards
- 2777 Views
- 3 replies
- 1 kudos
- 1 kudos
Do use the Databricks CLI or API 2.0 to manage secrets. Don't leave them in your notebooks for everyone to see, same applies for salting or hash strings.Secret management - Azure Databricks | Microsoft Docs
- 1 kudos
- 21723 Views
- 12 replies
- 11 kudos
Resolved! Is there any difference between performance of Python and SQL ?
I read somewhere that Python code is converted to SQL at the end. So is it true or there is any difference in performance while working with Scala, Python or SQL ?
- 21723 Views
- 12 replies
- 11 kudos
- 11 kudos
To add on the consideration of UDFs, try to consider using HOFs (Higher Order Functions) whenever possible first as there is a signifcant performance benefit as seen here.
- 11 kudos
- 1313 Views
- 0 replies
- 1 kudos
Structured streaming vs Confluent Kstream
For Ultra low latency customer facing App, I am curious on cost efficiency between Structured streaming and Kstream; which work better in terms of cost ? Though still achieving the ultra low latency and quality outcome. Appreciate any thoughts from p...
- 1313 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
