- 3939 Views
- 1 replies
- 0 kudos
How to enable "Create Vector Search Index" button in DB workspace?
How to enable "Create Vector Search Index" button in DB workspace?Following is the screenshot from the Microsoft Ignite 2023 Databricks presentation:
- 3939 Views
- 1 replies
- 0 kudos
- 0 kudos
The feature is in public preview only in some regions, you can check the available regions in the documentation here. In addition there are certain requirements, such as a UC enabled workspace and Serverless Compute enabled, you can check all requir...
- 0 kudos
- 6205 Views
- 5 replies
- 0 kudos
CONVERT_TIMEZONE issue in DLT
I can run a query that uses the CONVERT_TIMEZONE function in a SQL notebook. When I move the code to my DLT notebook the pipeline produces this error:Cannot resolve function `CONVERT_TIMEZONE`Here is the line: CONVERT_TIMEZONE('UTC', 'America/Phoen...
- 6205 Views
- 5 replies
- 0 kudos
- 0 kudos
Yes, the notebook is set to SQL and the convert_timezone function is within a select statement.
- 0 kudos
- 10321 Views
- 2 replies
- 1 kudos
Can we get the actual query execution plan programmatically after a query is executed? Apart from UI
Let's say i have run a query and it showed me results. we can find the respective query execution plan on the UI. Is there any way we can get that execution plan through programmatically or through API?
- 10321 Views
- 2 replies
- 1 kudos
- 1 kudos
You can obtain the query execution plan programmatically using the EXPLAIN statement in SQL. The EXPLAIN statement displays the execution plan that the database planner generates for the supplied statement. The execution plan shows how the table(s) r...
- 1 kudos
- 3985 Views
- 2 replies
- 4 kudos
Top Kudoed Author 🌟🤩🧑🎤
I recently saw a link to the Kudos Leaderboard for the Community Discussions. It has always been my hope and fantasy , ever since I was a little child that I would someday be the #1 Kudoed Author on Community Discusions on community.Databricks.com....
- 3985 Views
- 2 replies
- 4 kudos
- 12368 Views
- 5 replies
- 7 kudos
Incremental ingestion of Snowflake data with Delta Live Table (CDC)
Hello,I have some data which are lying into Snowflake, so I want to apply CDC on them using delta live table but I am having some issues.Here is what I am trying to do: @dlt.view() def table1(): return spark.read.format("snowflake").options(**opt...
- 12368 Views
- 5 replies
- 7 kudos
- 7 kudos
The CDC for delta live works fine for delta tables, as you have noticed. However it is not a full blown CDC implementation/software.If you want to capture changes in Snowflake, you will have to implement some CDC method on Snowflake itself, and read...
- 7 kudos
- 3674 Views
- 2 replies
- 0 kudos
New to PySpark
Hi all,I am trying to get the domain from an email field using below expression; but getting an error.Kindly help. df.select(df.email, substring(df.email,instr(df.email,'@'),length(df.email).alias('domain')))
- 3674 Views
- 2 replies
- 0 kudos
- 0 kudos
In your case, you want to extract the domain from the email, which starts from the position just after '@'. So, you should add 1 to the position of '@'. Also, the length of the substring should be the difference between the total length of the email ...
- 0 kudos
- 2411 Views
- 1 replies
- 0 kudos
Issue in inferring schema for streaming dataframe using json files
Below is the pileine design in databricks and it's not working out , kindly look on this and let me know whether it will work or not , I'm getting json files of different schemas from directory under the root directory and it read all the files using...
- 2411 Views
- 1 replies
- 0 kudos
- 0 kudos
Could you please share some sample of your dataset and code snippet of what you're trying to implement?
- 0 kudos
- 6943 Views
- 0 replies
- 0 kudos
Database: Delta Lake or PostgreSQL
Hey all,I am searching for a non-political answer to my database questions. Please know that I am a data newbie and litteraly do not know anything about this topic, but I want to learn, so please be gentle. Some context: I am working for an OEM that...
- 6943 Views
- 0 replies
- 0 kudos
- 8893 Views
- 2 replies
- 3 kudos
Resolved! Pros and cons of physically separating data in different storage accounts and containers
When setting up Unity Catalog, it is recommended by Databricks to figure out your data isolation model when it comes to physically separating your data into different storage accounts and/or contaners. There are so many options, it can be hard to be ...
- 8893 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @pernilak , Thanks for reaching out to Databricks Community! My name is Raphael, and I'll be helping out. Should all catalogs and the metastore reside in the same storage account (but different containers) Yes, Databricks recommends having o...
- 3 kudos
- 2537 Views
- 0 replies
- 0 kudos
New draft for every post I visit
When I visit my profile page, under the drafts section I see an entry for every post I visit in the discussions. Is this normal?
- 2537 Views
- 0 replies
- 0 kudos
- 2092 Views
- 1 replies
- 1 kudos
Databricks Web Editor's Cell like UI in local IDE
I want to have databricks related developement locally.There is extension that allows to run local python file on remote databricks cluster.But I want to have cell like structure that is present in databricks UI for python files in local IDE as well....
- 2092 Views
- 1 replies
- 1 kudos
- 1 kudos
@swapnilmd You can use VSCode extension for Databricks.https://docs.databricks.com/en/dev-tools/vscode-ext/index.html
- 1 kudos
- 6592 Views
- 2 replies
- 0 kudos
Databricks jdbc driver connectiion issue with apache solr
Hi,databricks jdbc version - 2.6.34I am facing the below issue with connecting databricks sql from apache solr Caused by: java.sql.SQLFeatureNotSupportedException: [Databricks][JDBC](10220) Driver does not support this optional feature.at com.databri...
- 6592 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks team recommended to set IgnoreTransactions=1 and autocommit=false in the connection string but that didn't resolve the issue .Ultimately I had to use solr update API for uploading documents
- 0 kudos
- 3121 Views
- 3 replies
- 1 kudos
[Memory utilization in Metrics Tab still display after terminate a cluster]
Hi All,Could you guys help me to check this?I run a cluster and then terminate that cluster but when i navigate to the Metrics tab of Cluster still see the Memory utilization show metrics.Thanks
- 3121 Views
- 3 replies
- 1 kudos
- 4622 Views
- 3 replies
- 0 kudos
databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace
databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace. The UI lists `hive_metastore`. It would be better if these two were consistent.
- 4622 Views
- 3 replies
- 0 kudos
- 2699 Views
- 0 replies
- 0 kudos
Both `hive_metastore` and `spark_catalog`?
HiIs it possible for a workspace to have both a `hive_metastore` catalog and a `spark_catalog` catalog?
- 2699 Views
- 0 replies
- 0 kudos
