- 825 Views
- 2 replies
- 2 kudos
Resolved! Deleting database tables from a catalog
Hello,I am exploring lakebase and sync the table to postgre catalog. When I try to drop a table, I got this errorof kind: Some(CATALOG_MANAGED_POSTGRESQL) is not supported. How to fix it? I have all the privileges. Thanks.Best regards
- 825 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @zyang! Could you please check if you’re following the steps outlined in the doc? First, you need to delete the synced table in Unity Catalog and then drop it from PostgreSQL using the DROP TABLE command.
- 2 kudos
- 966 Views
- 2 replies
- 0 kudos
Facing difficulty in connecting access db from databricks
Unable to connect Microsoft Access DB in Databricks, I tried with JDBC & Pyodbc connectors but I'm not able to read the file from accessdb. Please suggest or help to resolve this issue
- 966 Views
- 2 replies
- 0 kudos
- 0 kudos
Installed office365-REST-Python-Client==2.5.13UcanaccessDriver # File location and typefile_location = "/FileStore/tables/ABU.accdb"file_type = "accdb"import pyodbc# Connection string (Windows only, or if driver is available)conn_str = ( # Change t...
- 0 kudos
- 4432 Views
- 8 replies
- 3 kudos
Resolved! AutoLoader Pros/Cons When Extracting Data
Hi there, I am interested in using AutoLoader, but I'd like to get a bit of clarity if it makes sense in my case. Based on examples I've seen, an ideal use-case for AutoLoader is when we have some kind of landing path where we expect raw files to arr...
- 4432 Views
- 8 replies
- 3 kudos
- 3 kudos
I am in the position to have more and more use-cases for AutoLoader. This might be an obvious question but if I understand correctly, Christian's use-case without the landing stage could still be an auto-loader scenario if the extracts from source sy...
- 3 kudos
- 470 Views
- 1 replies
- 0 kudos
Fetching data from Databricks server with delta sharing limits to 100 records
Hi,I'm trying to fetch a table from a Databricks instance hosted in Azure using delta sharing python library. The delta sharing library always returns a dataframe of length 100 when fetching table data. I tested all the tables shared with me. The ins...
- 470 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @uddipak ,Maybe internally load_as_pandas has some default limit? Can you try to set limit explicitly?import delta_sharing client = delta_sharing.SharingClient("...") client.list_all_tables() # this works fine . . . delta_sharing.load_as_pandas(t...
- 0 kudos
- 5447 Views
- 2 replies
- 3 kudos
Resolved! Liquid clustering with boolean columns
Hi community Is it possible to use boolean columns as cluster keys for liquid clustering on Delta Tables? I've been trying to set a boolean column as cluster key since it's one of my most common queries when reading from the table. I'm getting the er...
- 5447 Views
- 2 replies
- 3 kudos
- 3 kudos
I saw that boolean columns are not supported by liquid clustering, too. Why? In any case:By now there is an error that can get raised called DELTA_CLUSTERING_COLUMNS_DATATYPE_NOT_SUPPORTED.There is also by now documentation of column datatypes that...
- 3 kudos
- 10173 Views
- 12 replies
- 3 kudos
Delay in files update on filesystem
Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If the...
- 10173 Views
- 12 replies
- 3 kudos
- 3 kudos
Similar issue where opening up the git interface can take 30+ seconds to show changes.
- 3 kudos
- 1851 Views
- 3 replies
- 4 kudos
Resolved! TABLE_OR_VIEW_ALREADY_EXISTS when using saveAsTable with append mode
I ran into an unexpected error while writing a DataFrame to a managed table df.write \ .mode("append") \ .option("overwriteSchema", "true") \ .option("mergeSchema", "true") \ .saveAsTable(bsr_sink_table_fqn)Instead of appending, Spark attemp...
- 1851 Views
- 3 replies
- 4 kudos
- 4 kudos
Thanks @szymon_dybczak and @WiliamRosa for the reply. I will probably keep mergeSchema then. If the options are incompatible or might cause issues I wish there was a warning of sorts.
- 4 kudos
- 3270 Views
- 5 replies
- 3 kudos
Resolved! Incrementalization issue in Materialized views
I am trying to implement the incremental updates to the Materialized views. The source is the Could you tell how to resolve the following issue? {  "planning_information": {    "technique_information": [      {        "maintenance_type": "MAINTENANCE...
- 3270 Views
- 5 replies
- 3 kudos
- 1058 Views
- 4 replies
- 5 kudos
Resolved! Lakeflow Connect SchemaParseException: Illegal character
Hi,i'm trying to setup Lakeflow Connect for SQL Server. The created gateway is failing with "org.apache.avro.SchemaParseException: Illegal character in: LN.FWH-ID"Unfortunately, don't have control over the source database to change the column names.I...
- 1058 Views
- 4 replies
- 5 kudos
- 5 kudos
Thanks for your response. I ended up solving this with federation. It's not ideal but given the data volumes and the minimal impact on source database, it works for us for now.
- 5 kudos
- 1408 Views
- 2 replies
- 0 kudos
Accessing Spark Runtime Metrics Using PySpark – Seeking Best Practices
Hi everyone,I’m currently working on a solution to access Spark runtime metrics for better monitoring and analysis of our workloads.From my research, I understand that this can be implemented using SparkListener, which is a JVM interface available in...
- 1408 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi saicharandeepb,How are you doing today? as per my understanding, since SparkListener is native to Scala/Java, getting detailed runtime metrics in PySpark can be tricky, but there are some workarounds. If you need deep metrics (like stage-level and...
- 0 kudos
- 5181 Views
- 2 replies
- 3 kudos
Resolved! HELP!
Hey Databricks Community, This may be a silly question but, are we able to share Databricks related job posts on here?
- 5181 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @TomFielding! If your question is about posting job openings in the Community, then the answer is No. The Community is intended for discussions and knowledge sharing around Databricks products.
- 3 kudos
- 2159 Views
- 11 replies
- 3 kudos
Resolved! Programmatically setting TAGs on VIEWs
Hi all,In order to achieve data stability in our Workspace, our IT team has given us access to the data through VIEWs on top of an ingestion schema.Now I want to provide metadata to VIEWs in form of TAGs (IT does not want to cover this used case). Th...
- 2159 Views
- 11 replies
- 3 kudos
- 3 kudos
Hi @szymon_dybczak and @WiliamRosa,I have used backticks from the very start (you'll see it if you re-check my original post).It is definitely a cluster issue as I am able to assign tags using serverless cluster. The reason I was using a dedicated on...
- 3 kudos
- 1498 Views
- 9 replies
- 6 kudos
Resolved! transformWithStateInPandas throws "Spark connect directory is not ready" error
Hello,we employ arbitrary stateful aggregations in our data processing streams on Azure Databricks, and would like to migrate from applyInPandasWithState to transformWithStateInPandas. We employ the Python API throughout our solution, and some of our...
- 1498 Views
- 9 replies
- 6 kudos
- 6 kudos
Update: This is working fine with earlier DBR versions, but the issue seems to occur specifically with DBR 17.1.I’ve flagged this behaviour with the internal team for further investigation.
- 6 kudos
- 4680 Views
- 7 replies
- 2 kudos
Databricks Key Vault Secret - Is it available in Databricks on AWS?
@Hubert-Dudek, I see your post regarding Key Value Secret handling via UI for Databricks on Azure.Is this feature available for Databricks on AWS as well?#Secrets #Scopes #DatabricksOnAWSThanksRamana
- 4680 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi, do you have any update on this topic? We are looking for the same thing. We are using Databricks (UC) on AWS and have a developer group that wants to implement 90-day rotation for an AWS IAM secret using Secrets Manager. I do not see anything i...
- 2 kudos
- 1918 Views
- 3 replies
- 0 kudos
serverless environment v3 JavaPackage object is not callable
run into this issue when use serverless environment v3JavaPackage object is not callable V2 works fine, any idea
- 1918 Views
- 3 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
