- 691 Views
- 3 replies
- 0 kudos
databricks structured streaming external table unity catalog
Hi everbody,I am facing a issue with spark structured steaming, with databricks on gcp. I use external table type but schema are managed schema. The code below in my notebook works very wellBut if i add column masking to target table, and rerun the n...
- 691 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @christian_chong, The error message you’re encountering indicates that the input path gs://table overlaps with other external tables or volumes. Specifically, it mentions conflicting tables or volumes: xxx.xxx.table and xxx.xxx.another_table. Y...
- 0 kudos
- 1466 Views
- 4 replies
- 2 kudos
Resolved! Create a SQL (Python) UDF in a Serverless SQL Warehouse using an external library
Hi There,Scenario at work is as described in the subject line:Is it possible to author a SQL (Python) scalar UDF IN A SQL SERVERLESS WAREHOUSE, which involves a library NOT included in any Databricks Runtime? And how would one go about it? (Documenta...
- 1466 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @cpelazza, Starting from Databricks Runtime 13.3 LTS and above, you can use the SQL CREATE FUNCTION statement to register scalar Python UDFs in the Unity Catalog.These Python UDFs allow you to write Python code and invoke it through a SQL functio...
- 2 kudos
- 540 Views
- 3 replies
- 0 kudos
Enable serverless in Delta Live Tables in Azure Databricks?
I'm trying to create a serverless DLT, as far I understood is in Public Review as listed here, Azure Databricks regions - Azure Databricks | Microsoft Learn. I've created a workspace in North Europe, but I cannot see the feature in preview.Could you ...
- 540 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Kaniz_Fatma,Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.Am I missing some detail?Thanks!
- 0 kudos
- 707 Views
- 2 replies
- 0 kudos
Access Control/Management Question
I have two workspaces made with the same account using same metastore and region, and I want the second workspace to be able to access only certain rows of tables from data held in the first workspace based on a user group condition. Is this possible...
- 707 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @hr959, Access control lists (ACLs) can be used to achieve fine-grained access control in Databricks. ACLs require the Premium plan or above. You can manage entitlements through the workspace admin settings page or APIs. For more details, refer to...
- 0 kudos
- 708 Views
- 2 replies
- 0 kudos
DLT fails with Queries with streaming sources must be executed with writeStream.start();
Hi guys!I am having an issue with passing the "streaming flow" between layers of the DLT.first layer "ETD_Bz" is passing through, but then "ETD_Flattened_Bz" is failing with "pyspark.errors.exceptions.captured.AnalysisException: Queries with streamin...
- 708 Views
- 2 replies
- 0 kudos
- 0 kudos
UPDATE: tried adding writeStream.start() like error suggested + as per other posts and ended up with following error/code: @dlt.table( name="ETD_Bz", temporary=False)def Bronze(): return (spark.readStream .format("delta") ...
- 0 kudos
- 674 Views
- 2 replies
- 0 kudos
PyTest working in Repos but not in Databricks Asset Bundles
Hello,I am trying to run PyTest from a notebook or python file that exists due to being deployed by a Databricks Asset Bundle (DAB).I have a repository that contains a number of files with the end goal of trying to run PyTest in a directory to valida...
- 674 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Kaniz_Fatma,Thankyou for your response. I am aware of what the error message means and that is exactly why I am requesting support. The same code deployed to two different locations in a workspace working differently is what I am trying to und...
- 0 kudos
- 462 Views
- 1 replies
- 0 kudos
Resolved! Coupon Code - Databricks Certified Data Engineer Professional.
Hello Team, I am planning to take "Databricks Certified Data Engineer Professional" certification exam.However, I have noticed that the exam fee is $200, which is a considerable expense for me at this time. I am reaching out to inquire if there are a...
- 462 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @RohitYadav1994, Thank you for sharing your concern with us! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hour...
- 0 kudos
- 4151 Views
- 5 replies
- 0 kudos
Resolved! Is LEGACY_SINGLE_USER_STANDARD no longer supported?
Based on [an answer in this community](https://community.databricks.com/s/question/0D58Y00009jfKM1SAM/error-no-metastore-assigned-for-the-current-workspace), we set "data security mode" to LEGACY_SINGLE_USER_STANDARD in a cluster policy which needed...
- 4151 Views
- 5 replies
- 0 kudos
- 934 Views
- 2 replies
- 1 kudos
Resolved! Enhancements: interact with DBFS breadcrumbs
Hi there,This is my first thread and it's a baby-foot step in the Databricks community, especially Data engineering section.am working in the community edition and I found this enhancement needed to be implemented: The need is to make the breadcrumbs...
- 934 Views
- 2 replies
- 1 kudos
- 1 kudos
Good feedback, thank you. we are actully looking to complelty revamp the databricks community edition and the experience will be much simpler. stay tuned.
- 1 kudos
- 362 Views
- 1 replies
- 1 kudos
Migration Azure to GCP
is there a place I could find the best practices to migrate Databricks from Azure to GCP?
- 362 Views
- 1 replies
- 1 kudos
- 1 kudos
If you want to migrate Databricks to Databricks, you should use Databricks Teraform Exporter for that job
- 1 kudos
- 487 Views
- 1 replies
- 0 kudos
Is it possible to use a finetuned model with genie spaces feature?
I was trying to explore Genie room, I observed that I may need to finetune the model. Can I finetuned exisiting model with the same UI?
- 487 Views
- 1 replies
- 0 kudos
- 0 kudos
No. Genie spaces uses a fine tuned specialised text-2-sql model. fine-tuning text to 2 SQL model is not an easy job, so I would not recommend you doing yourself unless your business requires it.
- 0 kudos
- 463 Views
- 2 replies
- 0 kudos
Trying to create external table in Hive Metastore
Receiving this error: KeyProviderException: Failure to initialize configuration for storage account adlspersonal.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyI used hive meta store to save my table %python spark....
- 463 Views
- 2 replies
- 0 kudos
- 0 kudos
not sure about that error, but try to do it using SQL, see if it will workcreate table hive_metastore.annual_enterprise_survey as select * from catalog.defaul.table
- 0 kudos
- 8249 Views
- 3 replies
- 1 kudos
Databricks public roadmap
Where can we see when what feature is expected? can you share the schedule?
- 8249 Views
- 3 replies
- 1 kudos
- 1 kudos
You need to subscribe to Databricks newsletter, and tune in for the Quarterly Product Roadmap Webinars
- 1 kudos
- 439 Views
- 1 replies
- 0 kudos
Resolved! How to integrate the CI/CD process with Databricks using Azure Devops on Catalog level.
How to integrate the CI/CD process with Databricks using Azure Devops on Catalog level instead of workspace level. I would like to understand the process if this is possible, given that if the catalog is used in different workspaces in same subscript...
- 439 Views
- 1 replies
- 0 kudos
- 0 kudos
CICD is not related to catalogs, it is related to environment (workspaces), there are lots of tutorials on youtube on how to setup Azure DevOps CICD to move assets from one workspace to another and start a job. you will need to use Databricks Plugin ...
- 0 kudos
- 280 Views
- 1 replies
- 0 kudos
Add option to skip or deactivate a task
It would be beneficial to have an option like a toggle to activate or deactivate a Task in the Job graph interface. This mainly helps to skip execution of a task and reactivate it as required. Currently there is no option to say I want this task to b...
- 280 Views
- 1 replies
- 0 kudos
- 0 kudos
Maybe just load the task with an empty notebook, and once decided just load the right notebook. not ideal but should do the job I guess
- 0 kudos