- 4156 Views
- 4 replies
- 1 kudos
Cluster policy type
Hi guys,I am creating a cluster policy through json. "runtime_engine": {"type": "fixed""value": "PHOTON"}When I run the above code... PHOTON option is getting enabled but graying out... What would I specify in type field so that the photon option sho...
- 4156 Views
- 4 replies
- 1 kudos
- 1 kudos
Hey, If you change the type to "allowlist" and provide "PHOTON" and "STANDARD" as options that should fix your issue.here is an example: "runtime_engine": {"type": "allowlist""value": ["PHOTON", "STANDARD"]"defaultValue": "PHOTON"}
- 1 kudos
- 2072 Views
- 1 replies
- 1 kudos
All my replies has been deleted and new one is being moderated (deleted) - why?
I noticed that when I Reply to post and trying to help solve community problem - my post are being either moderated (deleted) or just not being saved. Old post/replies has been deleted.Is there any reason for that?I kind of lost my will to participat...
- 2072 Views
- 1 replies
- 1 kudos
- 1 kudos
Currently, I'm also facing same issue, my comments are automatically deleted and also not receiving email for same, which is happening to specific post.
- 1 kudos
- 3168 Views
- 2 replies
- 2 kudos
Error accessing file from dbfs inside mlflow serve endpoint
Hi,I have mlflow model served using serverless GPU which takes audio file name as input and then file will be passed as parameter to huggiung face model inside predict method. But I am getting following errorHFValidationError(\nhuggingface_hub.utils....
- 3168 Views
- 2 replies
- 2 kudos
- 2 kudos
I have the same issue.I have a large file that I cannot access from an MLFlow service.Things I have tried (none of these work):Read-only from DBFS`dbfs:/myfolder/myfile.chroma` does not work`/dbfs/myfolder/myfile.chroma` does not workRead-only from U...
- 2 kudos
- 529 Views
- 0 replies
- 0 kudos
PowerBI reports migration from Synapse to Databricks sql
Hello Techie,Do someone please share detailed design consideration when migrating 250 powerBI reports from Azure Synapse, to Databricks (DB SQL).How to plan for cutover and strategy for PowerBI. what are the consideration to be done , is there any da...
- 529 Views
- 0 replies
- 0 kudos
- 4786 Views
- 5 replies
- 1 kudos
Query results in csv file include 'null' string for blank cell
After running a sql script, when downloading the results to a csv file, the file includes a null string for blank cells (see screenshot). Is ther a setting I can change to simply get empty cells instead?
- 4786 Views
- 5 replies
- 1 kudos
- 1 kudos
I understand, however this is more on CSV file format. Save your data in Delta format instead of CSV or text-based formats. Delta tables handle empty strings and NULL values more effectively, ensuring that empty strings are preserved during data ins...
- 1 kudos
- 899 Views
- 0 replies
- 0 kudos
How to Fetch Azure OpenAI api_version and engine Dynamically After Resource Creation via Python?
Hello,I am using Python to automate the creation of Azure OpenAI resources via the Azure Management API. I am successfully able to create the resource, but I need to dynamically fetch the following details after the resource is created:API Version (a...
- 899 Views
- 0 replies
- 0 kudos
- 5936 Views
- 6 replies
- 1 kudos
Materialized Views Without DLT?
I'm curious, is DLT *required* to use Materialized Views in Databricks? Is it not possible to create and refresh a Materialized view via a standard Databricks Workflow?
- 5936 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @ChristianRRL ,When creating a materialized view in Databricks, the data is stored in DBFS, cloud storage, or Unity Catalog volume. You can still create a materialized view by overwriting the same table each time, instead of using Append, Update, ...
- 1 kudos
- 4237 Views
- 4 replies
- 1 kudos
Purpose of DLT Table table_properties > quality:medallion
Hi there, silly question here but can anyone help me understand what practical purpose does labelling the table_properties with "quality":"<specific_medallion>"? For example: @Dlt.table( comment="Bronze live streaming table for Test data", name="...
- 4237 Views
- 4 replies
- 1 kudos
- 1 kudos
I'm with the same doubt @ChristianRRL, did you figured out something related to it?My doubt is to check if it's possible to apply any kind of access control based on this property.
- 1 kudos
- 2990 Views
- 6 replies
- 3 kudos
Resolved! Plotly Express not rendering in Firefox but fine in Safari
Using a basic example of plotly express i see no output in firefox but is fine in Safari. Any ideas why this may occur? import plotly.express as px import pandas as pd # Create a sample dataframe df = pd.DataFrame({ 'x': range(10), 'y': [2, 3, 5, 7...
- 2990 Views
- 6 replies
- 3 kudos
- 3 kudos
UPDATE: I reached out further to Databricks support and they have since deployed a fix. Works fine for me now!
- 3 kudos
- 7065 Views
- 8 replies
- 3 kudos
Unity catalog enabled workspace -Is there any way to disable workflow/job creation for certain users
Currently in unity catalog enabled workspace users with "Workspace access" can create workflows/jobs, there is no access control available to restrict users from creating jobs/workflows.Use case: In production there is no need for users, data enginee...
- 7065 Views
- 8 replies
- 3 kudos
- 3 kudos
@Lakshay Databricks offers a robust platform with a variety of features, including data ingestion, engineering, science, dashboards, and applications. However, I believe that some features, such as workflow/job creation, alerts, dashboards, and Genie...
- 3 kudos
- 3007 Views
- 3 replies
- 0 kudos
String to date conversion errors
Hi,I am getting data from CDC on SQL Server using Informatica which is writing parquet files to ADLS. I read the parquet files using DLT and end up with the date data as a string such as this'20240603164746563' I couldn't get this to convert using m...
- 3007 Views
- 3 replies
- 0 kudos
- 0 kudos
Checking on my current code, this is what I am using, which works for me because we don't use daylight savings time. from_utc_timestamp(date_time_utc, 'UTC-7') as date_time_local
- 0 kudos
- 23601 Views
- 5 replies
- 1 kudos
Insufficient privileges:User does not have permission SELECT on any file
Hello,after switching to "shared cluster" usage a python job is failing with error message: Py4JJavaError: An error occurred while calling o877.load. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User...
- 23601 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @GeKo The checkpoint directory, is that set on cluster level or how do we set that ? Can you please help me with this ?
- 1 kudos
- 1592 Views
- 1 replies
- 0 kudos
Databricks UC Data Lineage Official Limitations
Hi all.I have a huge data migration project using medallion architecture, UC, notebooks and workflows . One of the relevant requirements we have is to capture all data dependencies (upstreams and downstreams) using data lineage. I've followed all re...
- 1592 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @RobsonNLPT , Yes SQL CTE are supported by the data lineage service. You can track table that were created using CTEs. Here is an example that demonstrate the feature. CREATE TABLE IF NOT EXISTS mpelletier.dbdemos.menu ( recipe_id INT, ...
- 0 kudos
- 3188 Views
- 3 replies
- 1 kudos
Resolved! Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster
Hi,I need to ingest and transform historical climate data into a Delta table. The data is stored in .nc format (NetCDF). To work with this format, specific C libraries for Python are required, along with particular versions of Python libraries (e.g.,...
- 3188 Views
- 3 replies
- 1 kudos
- 2197 Views
- 5 replies
- 0 kudos
Service Principal Access to Users Directory in Databricks - Creating Git Folders
I am trying to automate the creation of git folders in user workspace directories triggered by GitHub feature branch creation. When developers create feature branches in GitHub, we want a service principal to automatically create corresponding git fo...
- 2197 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Brianhourigan, Can you please DIM your suggestions? I can add it to our internal AHA idea.
- 0 kudos
