Do you require for a Service Principal or a Group to have admin rights to allow automation or reduce the efforts in the process of adding the permission to each user. Solution For Service Principals: You need to be at least Workspace AdminYou can eit...
Executing databricks labs install ucx via databricks cli and running into 'WorkspaceInstaller is not supposed to be executed in databricks runtime.' How do I resolve it?
Databricks CLI is designed to work in your local terminal and connect to your workspace. The error is mentioning that the command is not allowed to be run in a Databricks cluster, as you are using the web terminal of a cluster in your workspace it w...
Initially, I tried registering while logging a model from mlflow and got:"Got an invalid source 'dbfs:/Volumes/compute_integration_tests/default/compute-external-table-tests-models/test_artifacts/1078d04b4d4b4537bdf7a4b5e94e9e7f/artifacts/model'. Onl...
I know, right! Look at the error message I posted:"Got an invalid source 'dbfs:/Volumes/compute_integration_tests/default/compute-external-table-tests-models/test_artifacts/1078d04b4d4b4537bdf7a4b5e94e9e7f/artifacts/model'. Only DBFS locations are cu...
Hi Team, Can you provide your thoughts on moving Databricks from Azure to GCP? What services are required for the migration, and are there any limitations on GCP compared to Azure? Also, are there any tools that can assist with the migration? Please ...
Hello Team, Adding to @sunnydata comments: Moving Databricks from Azure to GCP involves several steps and considerations. Here are the key points based on the provided context: Services Required for Migration:Cloud Storage Data: Use GCP’s Storage T...
Understanding Retrieval-Augmented Generation (RAG)Retrieval-Augmented Generation (RAG) is a cutting-edge AI paradigm that enhances traditional generative models by integrating real-time data retrieval. By combining retrieval and generation, RAG ensur...
Here's a demo using RAG LLM: https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/lakehouse-ai-deploy-your-llm-chatbot
Hi everyoneI'm new to databricks and to this platform. My organization just got started with databricks and we're looking to procure/purchase enterprise support to help with things like training, set up, maintenance and development of our warehousing...
Hi @victor_okrobodo, Please see: https://www.databricks.com/professional-services Let me know if you have any other questions.
What is the 'implicit' or 'by default' data model of databricks or unity catalog ? Is it Data Vault ?
Please suggest best ways to get databricks unity catalog data to Aws s3 or sagemaker. Data could be around 1gb in some tables and 20gb in others.currently sagemaker pipelines use data from s3 as batches in different parquet files. But now we would li...
You could try by using Delta Sharing with your provider as mentioned in doc https://docs.databricks.com/en/delta-sharing/set-up.html
As shown in below screenshots MERGE INTO command produces information about the result (num_affected_rows, num_updated_rows, num_deleted_rows, num_inserted_rows).Unable to get this information when the same query is being executed via JDBC client. Is...
Delta API can help you get these details. Reference - https://docs.databricks.com/en/delta/history.html#history-schema
Did the SparkUI training yesterday with Mark Ott, and I highly recommend it. It was super helpful and provided a lot of clarity around some of the vaguer terms and metrics, and some surprise penalties.In-memory partition size is the the main thing to...
Hi, I can't find this course... can you please share the full name of this course? Thanks in advance
We have Databricks UniForm format (iceberg) tables are present in azure data lake storage (ADLS) which has already integrated with Databricks unity catalog. How to read Uniform format tables using Databricks as a query engine?
Query Using Unity Catalog:SQL:sqlCopiar códigoSELECT * FROM catalog_name.schema_name.table_name;PySpark:pythonCopiar códigodf = spark.sql("SELECT * FROM catalog_name.schema_name.table_name") df.display()Direct Access by Path: If not using Unity Catal...
Hi,I have a single-node personal cluster with 56GB memory(Node type: Standard_DS5_v2, runtime: 14.3 LTS ML). The same configuration is done for the job cluster as well and the following problem applies to both clusters:To start with: once I start my ...
code:# Define the path for the staging Delta tablestaging_table_path = "dbfs:/user/hive/warehouse/staging_order_tracking"spark.sql( f"CREATE TABLE IF NOT EXISTS staging_order_tracking USING DELTA LOCATION '{staging_table_path}'" )Creating table in U...
I believe, using mount point only we can able to connect to our storage account - containers. If this is anti pattern by data bricks what is the way? Can you please explain what is external location of UC is it our local system folders or something n...
Hi, I saw a demo about "Databricks Apps" 2 months ago. I haven't used Databricks for about 3 months, and I recently recreated a Premium Workspace to try something out (I use Azure), however I can't find "Apps" when I click "New". How can I enable and...
Im having an issue to query registered tables on glue catalog thru databricks with the following error: AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view looker.ccc_data cannot be found.Verify the spelling and correctness of the schema a...
Can you specify the full catalog.schema.table? and also check the current schema SELECT current_schema();
