- 13948 Views
- 12 replies
- 9 kudos
Resolved! DLT pipeline MLFlow UDF error
I am running this notebook via the dlt pipeline in preview mode.everything works up until the predictions table that should be created with a registered model inferencing the gold table. This is the error: com databricks spark safespark UDFException...
- 13948 Views
- 12 replies
- 9 kudos
- 9 kudos
Hi @Feather Have you also tried specifying the version of the library as well?
- 9 kudos
- 3290 Views
- 1 replies
- 0 kudos
break production using a shallow clone
Hi,If you create a shallow clone using the latest LTS, and drop the clone using a SQL warehouse (either current or preview), the source table is broken beyond repair. Data reads and writes still work, but vacuum will remain forever broken. I've attac...
- 3290 Views
- 1 replies
- 0 kudos
- 0 kudos
To add to that: the manual does not state that this might happen
- 0 kudos
- 16898 Views
- 2 replies
- 1 kudos
Resolved! Access storage account with private endpoint
Hi, I need guidance on connecting Databricks (not VNET injected) to a storage account with Private Endpoint.We have a client who created Databricks with (public ip and not VNET Injected). It’s using a managed VNET in the Databricks managed resource g...
- 16898 Views
- 2 replies
- 1 kudos
- 1 kudos
No this is not possible because the workspace is not part of the virtual network and since than can not access the storage over it's private endpoint. It is all mentioned in de documentation:https://www.databricks.com/blog/2020/02/28/securely-access...
- 1 kudos
- 3431 Views
- 0 replies
- 0 kudos
Connect to storage with private endpoint from workspace EnableNoPublicIP=No and VnetInjection=No
We know that Databricks with VNET injection (our own VNET) allows is to connect to blob storage/ ADLS Gen2 over private endpoints and peering. This is what we typically do.We have a client who created Databricks with EnableNoPublicIP=No (secure clust...
- 3431 Views
- 0 replies
- 0 kudos
- 5092 Views
- 2 replies
- 0 kudos
Azure Databricks, migrating delta table data with CDF on.
We are on Azure Databricks over ADLS Gen2 and have a set of tables and workflows that process data from and between those tables, using change data feeds. (We are not yet using Unity Catalog, and also not Hive metastore, just accessing delta tables f...
- 5092 Views
- 2 replies
- 0 kudos
- 0 kudos
As it turns out, due to a misunderstanding, the responses from Azure support were answering a slightly different question (about Azure Table Storage instead of Delta Tables on Blob/ADLS Gen2), so we'll try there again. However, still interested in id...
- 0 kudos
- 4489 Views
- 1 replies
- 1 kudos
Resolved! Hive metastore table access control End of Support
Hello,We are using Databricks with Hive metastore and not Unity Catalog.We would like to know if there is any End of Support on Table Access Control with Hive as this link it states that it is legacy.https://docs.databricks.com/en/data-governance/tab...
- 4489 Views
- 1 replies
- 1 kudos
- 1633 Views
- 0 replies
- 0 kudos
Many dbutils.notebook.run interations in a workflow -> Failed to checkout Github repository Error
Hi all,I have a workflow that runs one single notebook with dbutils.notebook.run() and different parameters in one long loop.At some point, I do have random git erros in the notebook run:com.databricks.WorkflowException: com.databricks.NotebookExecut...
- 1633 Views
- 0 replies
- 0 kudos
- 6912 Views
- 4 replies
- 1 kudos
Resolved! First notebook in ML course fails with wrong runtime
Help! I'm trying to run this first notebook in the Scalable MachIne LEarning (SMILE) course.https://github.com/databricks-academy/scalable-machine-learning-with-apache-spark-english/blob/published/ML%2000a%20-%20Spark%20Review.pyIt fails on the first...
- 6912 Views
- 4 replies
- 1 kudos
- 1 kudos
it means your cluster type has to be a ML runtime.When you create a cluster in databricks, you can choose between different runtimes.These have different version (spark version), but also different types:For your case you need to select the ML menu o...
- 1 kudos
- 3422 Views
- 0 replies
- 0 kudos
Size of each partitioned file (partitioned by default)
When I try a describe detail I get the number of files the delta table is partitioned into. How can I check the size of each file of these files that make up my entire table ?Will I be able to query each partitioned file to understand how they have b...
- 3422 Views
- 0 replies
- 0 kudos
- 2226 Views
- 0 replies
- 0 kudos
Insufficient Permission when writing to AWS Redshift
I'm trying to write a table in AWS Redshift using the following code:try: (df_source.write .format("redshift") .option("dbtable", f"{redshift_schema}.{table_name}") .option("tempdir", tempdir) .option("url", url) ...
- 2226 Views
- 0 replies
- 0 kudos
- 2774 Views
- 1 replies
- 0 kudos
Streaming problems after Vaccum
Hi allTo read from a large Delta table, I'm using readStream but with a trigger(availableNow=True) as I only want to run it daily. This worked well for an intial load and then incremental loads after that.At some point though, I received an error fro...
- 2774 Views
- 1 replies
- 0 kudos
- 15091 Views
- 1 replies
- 1 kudos
Maintain the camelCase column names in the bronze layer, or is it advisable to rename column names
I am utilizing the Databricks autoloader to ingest files from Google Cloud Storage (GCS) into Delta tables in the bronze layer of a Medallion architecture. According to lakehouse principles, the bronze layer should store raw data Hi dear community,I...
- 15091 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @param_sen ,Navigating the nuances of naming conventions, especially when dealing with different layers in a lakehouse architecture, can be a bit of a puzzle. Your considerations are on point. If consistency across layers is a priority and downst...
- 1 kudos
- 12293 Views
- 3 replies
- 1 kudos
Resolved! What are the best practices in bronze layer regarding the column data types?
Hi dear community,When I used to work in the Hadoop ecosystem with HDS the landing zone was our raw layer, and we used to use AVRO format for the serialization of this raw data (for the schema evolution feature), only assigning names to columns but n...
- 12293 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi dear community,I am utilizing the Databricks autoloader to ingest files from Google Cloud Storage (GCS) into Delta tables in the bronze layer of a Medallion architecture. According to lakehouse principles, the bronze layer should store raw data wi...
- 1 kudos
- 1815 Views
- 0 replies
- 0 kudos
Function in juypter notebook 12x faster than in python script
Hello dear community,I wrote some ETL functions, e.g. to count the sessions until a conversion (see below). There for I load the data and then execute several small function for the feature generation.When I run the function feat_session_unitl_conver...
- 1815 Views
- 0 replies
- 0 kudos
- 4609 Views
- 1 replies
- 0 kudos
Run driver on spot instance
The traditional advice seems to be to run the driver on "on demand", and optionally the workers on spot. And this is indeed what happends if one chooses to run with spot instances in Databricks. But I am interested in what happens if we run with a dr...
- 4609 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your answer @Retired_mod ! Good overview, and I understand that "driver on-demand and the rest on spot" is a good generall advice. But I am still considering using spot instances for both, and I am left with two concrete questions:1: Can w...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
