- 2848 Views
- 1 replies
- 0 kudos
Consume updated data from the Materialized view and send it as append to a streaming table
Hello everyone! I'm using DLT and I'm pretty new to them. I'm trying to take the updates from a materialized view and send them to a streaming table as an append.For example, if I have a MV of 400 records, I want an append to be made to the streaming...
- 2848 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @WearBeard By default, streaming tables require append-only sources. The encountered error is due to an update or delete operation on the 'streaming_table_test'. To fix this issue, perform a Full Refresh on the 'streaming_table_test' table. You ca...
- 0 kudos
- 1882 Views
- 0 replies
- 0 kudos
Not able to create mount point in Databricks
HiI am trying to create mount point in Azure Databricks, but mount point creation is failing with below error messageDBUtils.FS Handler.mount() got an unexpected keyword argument 'extra_config'I am using following codedef setup_mount(storage_account_...
- 1882 Views
- 0 replies
- 0 kudos
- 1557 Views
- 1 replies
- 0 kudos
Exam got suspended without any reason
Hi Team,My Databricks Certified Associate Developer for Apache Spark 3.0 - Python exam got suspended on 7th March 2024I was there continuously in front of the camera and suddenly the alert appeared, and support person asked me to show the full table ...
- 1557 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @badari_narayan did you exam got rescheduled..i am also facing same issue my exam got suspemded
- 0 kudos
- 2746 Views
- 2 replies
- 0 kudos
Fail to install package dependency located on private pypi server during .whl installation
Hello,I recently switched from DBR 12.2 LTS to DBR 13.3 LTS and observed the following behavior:My goal is to install a python library from a .whl file. I am using the UI for this task (Cluster settings -> Libraries -> Install new -> 'Python Whl' as ...
- 2746 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Felix, I have run into a similar issue recently (my wheel needs a Git HTTPS redirect that's specified in the init script - but I can install it fine from inside a notebook).I wonder whether you found a solution (perhaps moving a more recent DBR v...
- 0 kudos
- 3043 Views
- 1 replies
- 0 kudos
How to ignore Writestream UnknownFieldException error
I have a parquet file that I am trying to write to a delta table:df.writeStream .format("delta") .option("checkpointLocation", f"{targetPath}/delta/{tableName}/__checkpoints") .trigger(once=True) .foreachBatch(processTable) .outputMode("append")...
- 3043 Views
- 1 replies
- 0 kudos
- 0 kudos
@mvmiller - Per the below documentation, The stream will fail with unknownFieldException, the schema evolution mode by default is addNewColumns. so, Databricks recommends configuring Auto Loader streams with workflows to restart automatically after s...
- 0 kudos
- 1598 Views
- 2 replies
- 0 kudos
[Bug] Orphan storage location
Hello,I'm not able to re-create an external location after removing its owner from Databricks Account. I'm getting the following error:Input path url 'abfss://foo@bar.dfs.core.windows.net/' overlaps with an existing external location within 'CreateEx...
- 1598 Views
- 2 replies
- 0 kudos
- 0 kudos
Your metastore admin can list all external locationsYour metastore admin can then drop the external location
- 0 kudos
- 7007 Views
- 2 replies
- 0 kudos
Resolved! Importing python to DLT - Not working with DLT Pipeline
Hello, we are trying to adapt our developments (notebook with delta tables), into Delta Live Tables Pipelines.We tried to import Python files that are very useful for data transformations (silver data cleaning, for example) :From the Cluster (run man...
- 7007 Views
- 2 replies
- 0 kudos
- 0 kudos
The solution is to import from Python but also add the python file in the Pipeline settings, in the list of source code.
- 0 kudos
- 3731 Views
- 3 replies
- 4 kudos
Parametrize the DLT pipeline for dynamic loading of many tables
I am trying to ingest hundreds of tables with CDC, where I want to create a generic/dynamic pipeline which can accept parameters (e.g table_name, schema, file path) and run the logic on it. However, I am not able to find a way to pass parameters to p...
- 3731 Views
- 3 replies
- 4 kudos
- 4 kudos
If you have different folders for each of your source tables, you can leverage python loops to naturally iterate over the folders.To do this, you need to create a create_pipeline function that has table_name, schema, path as your parameters. Inside t...
- 4 kudos
- 1881 Views
- 0 replies
- 0 kudos
Issue with applying ACL's in Unit catlog enabled workspace
We have been using Hive Metastore in Databricks workspaces and recently enabled Unity Catalog for one of the workspace. However, we are encountering issues while applying grants on databases. The system is complaining, stating that table access contr...
- 1881 Views
- 0 replies
- 0 kudos
- 12375 Views
- 1 replies
- 2 kudos
Delta Table on AWS Glue Catalog
I have set up Databricks cluster to work with AWS Glue Catalog by enabling the spark.databricks.hive.metastore.glueCatalog.enabled to true. However, when I create a Delta table on Glue Catalog, the schema reflected in the AWS Glue Catalog is incorrec...
- 12375 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, could you please refer to something or explain in more detail your point about querying Delta Lake files directly instead of through the AWS Glue catalog and why it was highlighted as a best practice?
- 2 kudos
- 1690 Views
- 1 replies
- 0 kudos
I would like to Create a schedule in Databricks that runs a job on 1st working day of every month
I would like to create a schedule in Databricks that runs a job on the first working day of every month (working days referring to Monday through Friday). I tried using Cron syntax but didn't have any luck. Is there any way we can schedule this in Da...
- 1690 Views
- 1 replies
- 0 kudos
- 0 kudos
@NDK_1 - Cron syntax won't allow the combination of day of month and day of week. you can try creating two different schedules - one for the first day, second day of the month and then add custom logic to check if it is an working day and then trigg...
- 0 kudos
- 44515 Views
- 8 replies
- 6 kudos
To read data from Azure Storage
Hi Team,May i know how to read Azure storage data in Databricks through Python.
- 44515 Views
- 8 replies
- 6 kudos
- 1151 Views
- 0 replies
- 0 kudos
Global ini file to reference Databricks-backed secrets (not Azure)
Is there a way to create a global ini file that will reference databricks-backed secrets? Not from Azure, we use databricks on AWS.
- 1151 Views
- 0 replies
- 0 kudos
- 15485 Views
- 2 replies
- 6 kudos
Resolved! CREATE TEMP TABLE FROM CTE
I have written a CTE in Spark SQL WITH temp_data AS ( ...... ) CREATE VIEW AS temp_view FROM SELECT * FROM temp_view; I get a cryptic error. Is there a way to create a temp view from CTE using Spark SQL in databricks?
- 15485 Views
- 2 replies
- 6 kudos
- 6 kudos
In the CTE you can't do a CREATE. It expects an expression in the form of expression_name [ ( column_name [ , ... ] ) ] [ AS ] ( query )where expression_name specifies a name for the common table expression.If you want to create a view from a CTE, y...
- 6 kudos
- 1129 Views
- 1 replies
- 0 kudos
Autoloader not detecting changes/updated values for xml file
if i update the value in xml then autoloader not detecting the changes.same for delete/remove column or property in xml. So request to you please help me to fix this issue
- 1129 Views
- 1 replies
- 0 kudos
- 0 kudos
It seems that the issue you're experiencing with Autoloader not detecting changes in XML files might be related to how Autoloader handles schema inference and evolution. Autoloader can automatically detect the schema of loaded XML data, allowing you...
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
