- 11027 Views
- 10 replies
- 0 kudos
Error in migration with UCX tool
Hey folks!I am facing an issue while migration the tables from Hive to UC using UCX tool; after completely running the setup and getting the assessment overview, we ran the following commands i.e. "databricks labs ucx create-table-mapping" but after...
- 11027 Views
- 10 replies
- 0 kudos
- 0 kudos
While running the UCX tool in workspace we are not able to access the tables which are created in hive metastore on default schema. When we run the migrate-table workflow we get the error that the tables are not accessible. We get the following error...
- 0 kudos
- 2426 Views
- 4 replies
- 0 kudos
Resolved! How to load PowerBI Dataset into databricks
Hi there, I would like to know how to load power bi dataset into databricks Thanks&Regards, zmsoft
- 2426 Views
- 4 replies
- 0 kudos
- 0 kudos
I don't think it's possible. While loading a table from DataBricks into a PowerBI dataset is possible, the opposite is not true.
- 0 kudos
- 4877 Views
- 3 replies
- 1 kudos
Error connecting Databricks Notebook using managed identity from Azure Data Factory
I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authenticat...
- 4877 Views
- 3 replies
- 1 kudos
- 1 kudos
Did you manage to solve it?I am getting the same error calling DBX REST API from ADF with System-assigned Managed Identity. The Databricks workspace is added a linked service and there no problem to run notebooks.
- 1 kudos
- 3303 Views
- 1 replies
- 2 kudos
Change Data Capture(CDC)
May I know what is a Change Data Capture ?
- 3303 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @subhas_hati ,Change data capture (CDC) is a data integration pattern that captures changes made to data in a source system, such as inserts, updates, and deletes. These changes, represented as a list, are commonly referred to as a CDC feed. You c...
- 2 kudos
- 16645 Views
- 9 replies
- 8 kudos
The log in function and password reset function in the community edition do not work
I want to register a databrick account. I already set up my account. I also receive the email to set my password. However, I cannot use my password to log in the community account. I can use it to log in my standard account. I also click the reset th...
- 16645 Views
- 9 replies
- 8 kudos
- 8 kudos
Hello, @lawrance Zhang​ - I wanted you to know that this isn't the first time we've heard of this recently. Thank you for opening a ticket. We've also escalated this to the team. We'll get there.
- 8 kudos
- 4768 Views
- 4 replies
- 2 kudos
Unable to stream from google pub/sub
I am trying to run below for subscribing to a pubsub but this code is throwing this exception java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2I have tried using all versions of https://mvnrepository.com/artifact/com.google...
- 4768 Views
- 4 replies
- 2 kudos
- 2 kudos
I see some issues from using pubsub as source. in the writeStream, both .foreach or .foreachBatch cannot work to be called when stream data arrives
- 2 kudos
- 2806 Views
- 2 replies
- 0 kudos
PySpark JSON read with strict schema check and mark the valid and invalid records based on the non-n
Hi,I have a use case where I have to read the JSON files from "/data/json_files/" location with schema enforced.For the completeness we want to mark the invalid records. The invalid records may be the ones where the mandatory field/s are null, data t...
- 2806 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sujitmk77, You have to ensure that valid records are processed while invalid records are marked appropriately, you can use the following PySpark code. This code reads the JSON files with schema enforcement and handles invalid records by marking t...
- 0 kudos
- 12791 Views
- 7 replies
- 6 kudos
How to connect Cognos 11.1.7 to Azure Databricks
We are trying to connect Cognos 11.1.7 to Azure Databricks, but no success.Can you please help or guide us how to connect Cognos 11.1.7 to Azure Databricks.This is very critical to our user community. Can you please help or guide us how to connect Co...
- 12791 Views
- 7 replies
- 6 kudos
- 6 kudos
Have anyone got the Simba JDBC driver going with CA 11.1.7? The ODBC driver works fine but i can't get the JDBC running.Regd's
- 6 kudos
- 14268 Views
- 10 replies
- 7 kudos
Resolved! How to obtain the server url for using spark's REST API
Hi,I want to access the stage and job information (usually available through Spark UI) through the REST API provided by Spark: http://<server-url>:18080/api/v1/applications/[app-id]/stages. More information can be found at following link: https://spa...
- 14268 Views
- 10 replies
- 7 kudos
- 7 kudos
Hi @Retired_mod and @menotron ,Thanks a lot; your solutions are working. I apologise for the delay, as I had some issue logging in.
- 7 kudos
- 3282 Views
- 1 replies
- 0 kudos
Resolved! how read through jdbc from postgres to databricks a particular data type
Hi Community,I need to load data from PostgreSQL into Databricks through JDBC without changing the data type of a VARCHAR[]column in PostgreSQL, which should remain as an array of strings in Databricks.Previously, I used psycopg2, and it worked, but ...
- 3282 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi community,Yesterday, I found a solution. This is to query through jdbc from postgres creating two columns that are manageable in databricks. Here the code: query = f"""(SELECT *, array_to_string(columns_to_export, ',') AS columns_to_export_strin...
- 0 kudos
- 3853 Views
- 7 replies
- 2 kudos
Some records are missing after window function
While loading data from one layer to another layer using pyspark window function, I noticed that some data is missing. This is happening if the data is huge. It's not happening for small quantity. Does anyone come across this issue before?
- 3853 Views
- 7 replies
- 2 kudos
- 2737 Views
- 3 replies
- 1 kudos
Failed to checkout Git repository: RESOURCE_DOES_NOT_EXIST: Attempted to move non-existing node
I'm having issues with checking out Git repo in Workflows. Databricks can access files from commit `a` but fails to checkout the branch when attempting to access commit `b`. The error occurs specifically when trying to checkout commit `b`, and Databr...
- 2737 Views
- 3 replies
- 1 kudos
- 1 kudos
I didn't do anything to fix it. Databricks support did something to my workspace to fix the issue.
- 1 kudos
- 4554 Views
- 1 replies
- 0 kudos
Accidentally deleted files via API
Hello,I’m hoping you might be able to help me.I have accidentally deleted some Workspace files via API (an Azure DevOps code deployment pipeline). I can’t see the files in my Trash folder – are they gone forever, or is it possible to recover them on ...
- 4554 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ohnomydata, Unfortunately files deleted via APIs or the Databricks CLI are permanently deleted and do not move to the Trash folder. The Trash folder is a UI-only feature, and items deleted through the UI can be recovered from the Trash within ...
- 0 kudos
- 2268 Views
- 2 replies
- 0 kudos
polars to natively read and write through adls
HI Everyone,Is there a way polars can directly read files from ADLS through abfss protocol.
- 2268 Views
- 2 replies
- 0 kudos
- 0 kudos
@pradeepvatsvk wrotae:HI Everyone,Is there a way polars can directly read files from ADLS through abfss protocol.Hello @pradeepvatsvk,Yes, Polars can directly read files from Azure Data Lake Storage (ADLS) using the ABFS (Azure Blob Filesystem) prot...
- 0 kudos
- 1841 Views
- 3 replies
- 0 kudos
Managed Delta Table corrupted
Hey guys,Recently, we have add some properties to our delta table and after that, the table shows error and we cannot do anything. The error is that: (java.util.NoSuchElementException) key not found: spark.sql.statistics.totalSizeI think maybe this i...
- 1841 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Rafael-Sousa, Could you please raise a support case for this, to investigate this further? help@databricks.com
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
